Generating an augmented view of a location of interest

ABSTRACT

One embodiment of the present invention sets forth a technique for generating an augmented view of a location of interest for a user. The technique includes determining an image capture location based on the location of interest, and determining an image capture direction based on the image capture location. The technique further includes receiving an image associated with the image capture location and the image capture direction. The technique further includes processing the image based on a line of sight associated with a location of the user to generate a processed image and displaying the processed image to the user. Advantageously, the disclosed techniques enable a user to see through obstructions in the surrounding environment to view a location of interest.

BACKGROUND

Field of the Embodiments of the Invention

Embodiments of the present invention generally relate to augmentedreality and, more specifically, to generating an augmented view of alocation of interest for a user.

Description of the Related Art

Navigation systems are widely used in a variety of applications todetermine the location of a user and to provide directions to enable theuser to navigate to a particular destination. A navigation systemtypically includes a location receiver that receives signals (e.g.,geolocation signals) from one or more sources, including GlobalPositioning System (GPS) satellites, a wireless access point, agyroscope, an inertial sensor, a compass, and the like. Based on thelocation signals received from the location receiver, navigationinformation is provided to the user.

In addition to providing a user with directions to a particulardestination, current navigation systems often enable the user to viewimages of the navigation route and/or destination. For example, somenavigation systems allow the user to preview a navigation route byviewing database images of the streets on which the user will betraveling. Thus, the user is able to become more familiar with theenvironment in which he or she will be traveling. However, viewingdatabase images of the navigation route is often impractical once theuser has embarked on a trip. For example, attempting to view databaseimages via a mobile device, such as a smartphone, while walking ordriving to a destination may distract the user, interfering with theuser's ability to safely navigate his or her surroundings. As such, ifthe user becomes lost or wishes to view images of the navigation routeor destination, he or she may have to stop walking or driving to operatehis or her mobile device.

To address these shortcomings, some navigation systems implementaugmented reality features, such as a heads-up display (HUD) that isoverlaid with the user's field of view. For example, information such asturn-by-turn directions, street names, and the remaining distance to adestination may be displayed in the user's field of view, enabling theuser to simultaneously view navigation information and pay attention tohis or her surroundings. Additionally, some navigation systems attemptto overlay relevant information on objects in the surrounding area, suchas by overlaying a street name, street number, and/or business names ona particular building. However, the overlay generated by currentaugmented reality systems is unable to provide the user with relevantinformation when an obstruction in the environment, such as a buildingor an automobile, is blocking the user's field of view. For example, ifthe user's view of a destination is obstructed by a building or anautomobile in the surrounding area, the user may be unable to view theaugmented reality information until the user has already reached thedestination.

As the foregoing illustrates, improved techniques for providing a userwith relevant navigation information would be useful.

SUMMARY

One embodiment of the present invention sets forth a method forgenerating an augmented view of a location of interest for a user. Themethod includes determining an image capture location based on thelocation of interest, and determining an image capture direction basedon the image capture location. The method further includes receiving animage associated with the image capture location and the image capturedirection. The method further includes processing the image based on aline of sight associated with a location of the user to generate aprocessed image and displaying the processed image to the user.

Further embodiments provide, among other things, a system and anon-transitory computer-readable medium configured to carry out methodsteps set forth above.

Advantageously, the disclosed techniques enable a user to see throughobstructions in the surrounding environment to view a location ofinterest. Additionally, the disclosed techniques enable the user tocontinue to pay attention to his or her real-time surroundings whileviewing the location of interest. Accordingly, by overlaying an image ofthe location of interest with the user's real-time field of view,navigation efficacy and user safety are increased.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIGS. 1A-1C are conceptual diagrams illustrating a technique forcompositing an unobstructed image of a location of interest with areal-time view to generate an augmented view of the location ofinterest, according to various embodiments of the present invention;

FIG. 2 is a conceptual diagram illustrating a technique for determiningan image capture location and an image capture direction when generatingan augmented view of a location of interest, according to variousembodiments of the present invention;

FIGS. 3A and 3B are conceptual diagrams illustrating a technique forgenerating an augmented view of a location of interest that isobstructed by a street corner, according to various embodiments of thepresent invention;

FIG. 4 is a conceptual diagram illustrating a technique for determiningimage capture location(s) and image capture direction(s) when generatingan augmented view of a location of interest that is obstructed by astreet corner, according to various embodiments of the presentinvention;

FIG. 5 is a flow diagram of method steps for generating an augmentedview of a location of interest for a user, according to variousembodiments of the present invention;

FIGS. 6A and 6B are conceptual diagrams illustrating a technique forgenerating a magnified view of a location of interest, according tovarious embodiments of the present invention;

FIG. 7 is a conceptual diagram illustrating a technique for determiningan image capture location based on a zoom level when generating amagnified view of a location of interest, according to variousembodiments of the present invention;

FIG. 8 is a flow diagram of method steps for generating a magnified viewof a location of interest 110 for a user, according to variousembodiments of the present invention; and

FIG. 9 is a block diagram illustrating a computing device configured toimplement one or more aspects of the present invention.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the embodiments of the presentinvention. However, it will be apparent to one of skill in the art thatthe embodiments of the present invention may be practiced without one ormore of these specific details.

FIG. 1A-1C are conceptual diagrams illustrating a technique forcompositing an unobstructed image 112 of a location of interest 110 witha real-time view 116 to generate an augmented view of the location ofinterest 110, according to various embodiments of the present invention.As shown in FIG. 1A, when a user is traveling, objects in thesurrounding environment may obstruct the user's view, interfering withthe user's ability to see locations of interest 110 such as signs,streets, buildings, landmarks, and the like. For example, in FIG. 1A, anobstruction 114 is blocking the user's view of a location of interest110 (e.g., a building) to which the user is traveling. As a result, theuser may not see the location of interest 110 in time to change trafficlanes and/or turn onto the appropriate street. Further, the user mayunnecessarily expend attentional resources attempting to visually locatea location of interest 110 that is blocked by an obstruction 114,decreasing the user's ability to pay attention to pedestrians and othervehicles in the surrounding environment.

In order to enable the user to view a location of interest 110 that isbehind an obstruction 114, an image 112 of the location of interest 110may be acquired from a database and/or from a camera that is positionedin the surrounding environment. The image 112 of the location ofinterest 110 may then be displayed in the user's field of view, enablingthe user to view the location of interest 110 while simultaneouslyviewing his or her real-time surroundings. In various embodiments, theimage 112 of the location of interest 110 may be shown to the user on adisplay 120. The display 120 may be disposed on and/or projected ontoany surface of the user's vehicle, including the windows (e.g., sidewindows, windshield, rear window, etc.), the mirrors (e.g., rearviewmirror, side view mirrors, etc.), and/or a display screen. In someembodiments, the display 120 is a heads-up display (HUD) that issubstantially transparent, enabling the user to view the location ofinterest 110 while also viewing his or her real-time surroundings. Amongother things, displaying images 112 on a transparent display 120 allowsthe location of interest 110 to be composited with the user's real-timeview 116, creating a substantially continuous view of the surroundingenvironment, as shown in FIG. 1B. Additionally, as shown in FIG. 1C,displaying the images 112 on a transparent display allows the opacity ofthe images 112 to be varied such that the user is able to view theobstruction 114 (e.g., the bus) in the same region of the display 120 inwhich the image 112 of the location of interest 110 is being displayed.Moreover, the opacity of the image 112 of the location of interest 110may be varied relative to the user's real-time view 116 by capturingreal-time images of the user's line of sight, compositing the real-timeimages with the images 112 of the location of interest 110, anddisplaying the composited images on the display 120. Such techniques aredescribed below in further detail in conjunction with FIG. 2.

FIG. 2 is a conceptual diagram of a navigation system 200 for displayingan augmented view of a location of interest 110 to a user, according tovarious embodiments of the present invention. As shown, the navigationsystem 200 may include a display 120, a camera 210, and a trackingdevice 220. The camera 210 may be configured to capture images of theuser's surroundings, such as the real-time view 116 described above. Thecamera 210 may be located in a position that enables it to captureimages from the user's point of view. For example, the camera 210 may belocated above, below, or to the side of the user's line of sight. Insome embodiments, the camera 210 may be mounted on a vehicle in whichthe user is traveling and/or on the user himself or herself.

In operation, the navigation system 200 receives a location of interest110. The location of interest 110 may be inputted by the user,determined by the navigation system 200, and/or received from asecondary device, such as a database. The location of interest 110 mayinclude a street address, street number, name, city, state, coordinates,and/or any other information that enables the navigation system 200 todetermine the identity and/or the location of interest 110. Thenavigation system 200 may further determine the location of the userrelative to the location of interest 110. The location of the user maybe determined using any type of geolocation device, including GlobalPositioning System (GPS) devices, wireless communication devices,time-of-flight devices, radio frequency (RF) devices, optical devices,and the like.

After receiving the location of interest 110, the navigation system 200receives an image 112 associated with the location of interest 110 froma database or from a camera that is located in the surrounding area. Insome embodiments, the camera is proximate to the location of interest110 such that the camera can capture an unobstructed image 112 of thelocation of interest 110 and transmit the image to the navigation system200. The database may be a navigation database, such as Google® StreetView, that includes images of various streets and paths captured frommultiple locations and from multiple directions.

The image 112 of the location of interest 110 received from the databaseand/or captured by a camera is associated with an image capturelocation. The image capture location may be selected to provide the userwith an image 112 of the location of interest 110 that is of anappropriate perspective (e.g., similar to the user's perspective) and isrelatively unobstructed by objects in the surrounding environment. Invarious embodiments, the image capture location may be determined basedon one or both of the location of interest 110 and the location of theuser. For example, if the location of interest 110 is a place ofbusiness that is on a street along which the user is walking or driving,then the image capture location may be positioned between the user andthe location of interest 110, for example, in the line of sight betweenthe user and the location of interest 110. Thus, if the user isapproaching the location of interest 110 from the south, then the imagecapture location may be positioned to the south of the location ofinterest 110—between the location of the user and the location ofinterest 110—in order to provide an image 112 having the properperspective.

The image 112 of the location of interest 110 received from the databaseand/or captured by a camera also may be associated with an image capturedirection. As described above with respect to the image capturelocation, the image capture direction may be selected to provide theuser with an image 112 of the location of interest 110 that is of anappropriate perspective (e.g., similar to the user's perspective). Theimage capture direction may be determined based on one or more of thelocation of interest 110, the location of the user, and the imagecapture location. For example, if the image capture location ispositioned between the user and the location of interest 110, then theimage capture direction may be selected to substantially match thedirection of the line of sight from the user to the location of interest110. Thus, if the user is approaching the location of interest 110 fromthe south, then the image capture direction may be substantiallynorthward in order to provide an image 112 having the properperspective. Alternatively, if the image capture location is notpositioned substantially along the line of sight between the user andthe location of interest 110, for example, if an image 112 associatedwith the image capture location described above is not available, thenthe image capture direction may be selected to compensate for thedifference in perspective between the image capture location and thelocation of the user. In addition, the image 112 itself may be processed(e.g., via a transform) to compensate for the difference in perspectivebetween the image capture location and the location of the user, as isdescribed below in further detail.

In various embodiments, the image capture location and/or image capturedirection may be determined by comparing one or more images 112 of thelocation of interest 110 that are stored in a database or captured byone or more cameras in the surrounding area to the real-time view 116 ofthe user. For example, the navigation system 200 may compare one or morerelevant portions of the images 112 of the location of interest110—captured at one or more image capture locations and/or image capturedirections—to corresponding portions of images of the real-time view 116that are captured by the camera 210. The navigation system 200 may thendetermine which image(s) are the closest match to the real-time view116. These image(s) 112 may then be processed and displayed to the useron the display 120, such as by applying a transform and compositing theimage(s) 112 with the real-time view 116.

Prior to displaying the image 112 of the location of interest 110 to theuser on the display 120, the image 112 may be processed. Processing mayinclude, for example, applying a transform to the image 112, resizingthe image 112, cropping the image 112, and/or combining multiple images112 into a single image 112 such that the image 112 and the real-timeview 116 are substantially continuous when the image 112 is displayed tothe user. In addition, the opacity of the image 112 may be modified tomore effectively blend the image 112 with the real-time view 116 and/orto enable the user to simultaneously view both the image 112 andcorresponding portion(s) of the real-time view 116 (e.g., theobstruction 114). Further, in some embodiments, the image 112 of thelocation of interest 110 may be composited with images of the real-timeview 116, and the resulting composited image(s) may be displayed to theuser on the display 120. Compositing the image 112 of the location ofinterest 110 with images of the real-time view 116 may be useful whenthe display 120 is not in the line of sight between the user and thelocation of interest 110. In other embodiments, only the image 112(e.g., the processed image 112) of the location of interest 110, butimages of the real-time view 116, is displayed on the display 120, suchas on a vehicle window (e.g., the windshield).

The tracking device 220 may include, without limitation, one or morecameras and/or sensors configured to determine the position and/or eyegaze direction 223 (e.g., horizontal angle and/or vertical angle) of theuser. Images acquired by the camera(s) and/or sensor(s) may be analyzedby a computing device included in the tracking device 220 and/orassociated with the navigation system 200. For example, as shown in FIG.2, a camera may be configured to acquire images of a user's eyes orface. The images may then be analyzed to determine the position of theuser relative to the display 120 and/or the direction in which the useris looking.

The position of the user and/or the eye gaze direction 223 may be usedto determine the location of interest 110, the image capture location,and/or the image capture direction. For example, the user may activatean augmented view function of the navigation device 200 and then look inthe direction of a location of interest 110. In response, the trackingdevice 220 may detect the user's eye gaze direction 223 and transmit theeye gaze direction 223 to a computing device associated with thenavigation system 200. The computing device may then use the eye gazedirection 223 to determine the location of interest 110 at which theuser is looking and to select an appropriate image capture locationand/or image capture direction. Images 112 associated with the imagecapture location and/or image capture direction may then be displayed tothe user. Further, the position of the user relative to the display 120(e.g., the position of the user within the vehicle) may be used todetermine and/or refine the image capture location and/or image capturedirection. For example, if the user moves to the left or right withinthe vehicle, then the image capture location and/or image capturedirection may be modified so that the image 112 of the location ofinterest 110 displayed to the user more accurately reflects the user'sperspective. Moreover, the position of the user relative to the display120 may be used to determine how to process the image 112 (e.g., viatransform, zooming, cropping, etc.) prior to displaying the image 112 tothe user.

Once the navigation system 200 receives an image 112 of the location ofinterest 110, the image 112 may be processed to compensate for thedifference in the height at which the image 112 was captured and theheight of the user's line of sight. For example, if the image 112 isacquired from a database, such as Google® Street View, then the image112 may have been captured at a height that is above the user's line ofsight. Accordingly, a transform may be applied to the image 112 tocompensate for the angle 222 between the line of sight of the user andthe line of sight of the camera (e.g., Google® Street View camera).

In some embodiments, the navigation device 200 is worn by the user. Forexample, the navigation device 200 may be a headset or pair of glassesworn by the user. In such embodiments, the camera 210, tracking device220, and/or display 120 could be positioned in the navigation device 200proximate to the user's eyes to determine the eye gaze direction 223 andprovide the user with images 112 of the location of interest 110. Forexample, one or more cameras may be positioned in a headset or pair ofglasses near the user's eyes.

Under certain circumstances, a user of the navigation device 200 maywish to view a location of interest 110 that is on the other side of astreet corner. For example, when a user is traveling, the user may wishto view what is on a particular street, prior to reaching the street, inorder to determine whether he or she should turn down the street.Accordingly, some of the techniques described above may be applied toprovide the user with images 112 of the buildings and/or other locationsof interest 110 that are around a street corner. Such implementationsare discussed below in conjunction with FIGS. 3A-5.

FIGS. 3A and 3B are conceptual diagrams illustrating a technique forgenerating an augmented view of a location of interest 110 that isobstructed by a street corner, according to various embodiments of thepresent invention. As shown, when approaching a street corner, a usermay wish to view the buildings, landmarks, signs, etc. that are beingobstructed by the street corner. Accordingly, the user may activate anaugmented view function of the navigation device 200 and input therelevant location of interest 110 (e.g., via a keyboard, voice command,eye gaze direction 223, and the like). In response, the navigationsystem 200 may determine an image capture location and/or image capturedirection based on the location of the user relative to the location ofinterest 110, as described below in further detail in conjunction withFIG. 4. The navigation system 200 may then receive an image 112 of thelocation of interest 110 based on the image capture location and/orimage capture direction and perform optional processing on the image112. The image 112 is then displayed to the user on the display 120. Forexample, as shown in FIG. 3B, the image 112 of a location of interest110 that is on the other side of a street corner may be displayed to theuser on a transparent display 120, such as a car window, enabling theuser to view the real-time view 116 in conjunction with the image 112.

FIG. 4 is a conceptual diagram illustrating a technique for determiningimage capture location(s) 415 and image capture direction(s) whengenerating an augmented view of a location of interest 110 that isobstructed by a street corner, according to various embodiments of thepresent invention. As shown, one or more images 112, each associatedwith a different image capture location 415 and/or image capturedirection, may be acquired from a database and/or from a camera that isproximate to the location of interest 110. When multiple images 112 areacquired, the images 112 may be combined to form a single, continuousimage 112 that is displayed to the user. Additionally, the image(s) 112may be combined with the real-time view 116 of the user. For example, asshown, images of real-time view 116-1 and real-time view 116-2,associated with the location of the user 410, may be combined with animage 112-1 captured at image capture location 415-1 and an image 112-2captured at image capture location 415-2 to form a continuous image thatis displayed to the user. In other embodiments, the continuous image maybe generated by combining only image 112-1 and image 112-2. Theresulting image 112 may then be processed (e.g., according to theperspective of the user) and displayed in the appropriate position inthe field of view of the user so that the image 112 appearssubstantially continuous with real-time view 116-1 and real-time view116-2.

In addition to compositing the real-time view(s) 116 with the images 112of the location of interest 110, the real-time view(s) 116 may be usedto determine which image capture location(s) and/or image capturedirection(s) provide the closest match to the real-time view 116associated with the location of the user 410. For example, a computingdevice associated with navigation system 200 may compare one or moreportions of the real-time view 116 (e.g., real-time view 116-1 and116-2) to one or more corresponding portions of images 112 acquired froma database or camera to determine which image(s) 112 most closely matchthe perspective of the user. Images 112 that provide an appropriatelevel of continuity with the real-time view 116 may then be processedand displayed to the user.

FIG. 5 is a flow diagram of method steps for generating an augmentedview of a location of interest for a user, according to variousembodiments of the present invention. Although the method steps aredescribed in conjunction with the systems of FIGS. 1-4, persons skilledin the art will understand that any system configured to perform themethod steps, in any order, falls within the scope of the presentinvention.

As shown, a method 500 begins at step 510, where the navigation system200 receives a location of interest 110. The location of interest 110may be inputted by the user, determined by the navigation system 200,and/or received from a secondary device, such as a database. At step515, the navigation system 200 determines the location of the user. Thelocation of the user may be determined using any type of geolocationdevice, including Global Positioning System (GPS) devices, wirelesscommunication devices, time-of-flight devices, radio frequency (RF)devices, optical devices, and the like.

Next, at step 520, the navigation system 200 determines an image capturelocation based on the location of interest 110 and the location of theuser. At step 525, the navigation system 200 may determine an imagecapture direction based on the location of interest 110, the location ofthe user, and/or the image capture location. Then, at step 530, thenavigation system 200 receives an image 112 of the location of interest110 associated with the image capture location and the optional imagecapture direction.

At step 540, the navigation system 200 and/or a computing deviceassociated with the navigation system 200 may process the image 112.Processing may include, for example, applying a transform to the image112, resizing the image 112, cropping the image 112, combining multipleimages 112 into a single image 112, and the like. At step 550, theprocessed image 112 is displayed to the user on a display 120.

At step 560, the navigation system 200 determines whether there has beena change to the location of the user. If there has been a change to thelocation of the user, then the method 500 returns to step 515, where thelocation of the user is determined. If there has not been a change tothe location of the user, then the method 500 proceeds to step 570. Atstep 570, the navigation system 200 determines whether there has been achange to the location of interest 110. If there has been a change tothe location of interest 110, then the method 500 returns to step 510,where a different location of interest 110 is received. If there has notbeen a change to the location of interest, then the method 500 ends.

In addition to generating an augmented view of an obstructed location ofinterest 110 for a user, the navigation system 200 may be configured togenerate an augmented view of a location of interest 110 that isrelatively unobstructed, but which is relatively far away from the user.Such implementations are illustrated in FIGS. 6A and 6B, which areconceptual diagrams illustrating a technique for generating a magnifiedview of a location of interest 110, according to various embodiments ofthe present invention. As shown in FIG. 6A, when a user is approachingan unobstructed location of interest 110, the user may wish to moreeasily view details associated with the location of interest 110.Accordingly, the navigation system 200 may acquire an image 112 of thelocation of interest 110 and display the image 112 on the display 120according to a zoom level specified by the user or determined by thenavigation system 200. The resulting image 112 of the location ofinterest 110 is shown in FIG. 6B. As shown, by generating a magnifiedimage of the location of interest 110, the user may more easily viewdetails associated with the location of interest 110.

The magnified image 112 displayed to the user may be acquired from adatabase, as described above, or from a camera that is proximate to thelocation of interest 110. Such implementations are described in furtherdetail below in conjunction with FIG. 6. Alternatively, the magnifiedimage 112 displayed to the user may be acquired by a camera (e.g.,camera 210) associated with the navigation system 200 and/or coupled tothe user or the user's vehicle. For example, the camera 210 associatedwith the navigation system 200 may provide a magnified image 112 of thelocation of interest 110 via an optical zoom or digital zoom function.In other embodiments, an image 112 captured by the camera 210 may bemagnified by a computing device associated with the navigation system200. The magnified image 112 may then be displayed to the user in anappropriate position relative to the real-time view 116.

FIG. 7 is a conceptual diagram illustrating a technique for determiningan image capture location based on a zoom level when generating amagnified view of a location of interest 110, according to variousembodiments of the present invention. As shown, the image capturelocation 415 selected by the navigation system 200 may be based on azoom level specified by the user or selected by the navigation system200. For example, image capture location 415-3 may be selected by thenavigation system 200 if the user inputs a relatively low zoom level,such as a zoom level of 2×, and image capture location 415-5 may beselected by the navigation system 200 if the user inputs a relativelyhigh zoom level, such as a zoom level of 24×. In various embodiments,the user may specify the zoom level at which the location of interest110 is to be displayed by issuing a voice command to the navigationsystem 200 or by inputting the zoom level on a keyboard, switch, knob,or other input device.

In addition to selecting an appropriate image capture location based onthe zoom level, in some embodiments, an image capture direction may beselected to provide an image 112 having the appropriate perspective, asdescribed above in conjunction with FIGS. 1A-5. Moreover, the image 112acquired for a given zoom level may be processed according to thetechniques described above to compensate for a difference in perspectivebetween the image capture location and the location of the user 410.Further, although the images 112 in FIG. 6 are illustrated as beingcaptured by vehicles, such as a Google® Street View vehicle or a vehiclepositioned in the surrounding environment (e.g., via vehicle-to-vehicle(V2V) communication), the images 112 may be captured from other objects(e.g., surveillance cameras on poles, street signs, buildings, etc.) andlocations in the surrounding environment.

In some embodiments, the techniques for viewing a location of interest110 that is obstructed by an object and/or a street corner, describedabove in conjunction with FIGS. 1A-5, may be combined with thetechniques for displaying a magnified image 112 of a location ofinterest 110 to a user, described above in conjunction with FIGS. 6A-7.For example, instead of display an image 112 of the location of interest110 to generate a continuous image from the perspective of the user, theimage 112 may be magnified to enable the user to more easily viewdetails associated with the location of interest 110. As describedabove, magnification of the image 112 may be specified by the userand/or selected by the navigation system to provide zoom levels rangingfrom no zoom—such that the image 112 is substantially continuous withthe real-time view 116—to a high level of zoom (e.g., 12×, 24×, orhigher).

FIG. 8 is a flow diagram of method steps for generating a magnified viewof a location of interest for a user, according to various embodimentsof the present invention. Although the method steps are described inconjunction with the systems of FIGS. 1-4 and 6-7, persons skilled inthe art will understand that any system configured to perform the methodsteps, in any order, falls within the scope of the present invention.

As shown, a method 800 begins at step 810, where the navigation system200 receives a location of interest 110. At step 812, the navigationsystem 200 determines the location of the user. At step 814, thenavigation system 200 receives a zoom level. The zoom level may beselected by the navigation system 200 (e.g., based on an algorithm), orthe user may specify the zoom level by issuing a voice command to thenavigation system 200 or by inputting the zoom level on a keyboard,switch, knob, or other input device.

Next, at step 820, the navigation system 200 determines an image capturelocation and optional image capture direction based on the location ofinterest 110, the location of the user, and/or the zoom level. At step830, the navigation system 200 receives an image 112 of the location ofinterest 110 associated with the image capture location and the optionalimage capture direction. Then, at step 840, the navigation system 200and/or a computing device associated with the navigation system 200 mayprocess the image 112. At step 850, the processed image 112 is displayedto the user on a display 120.

At step 860, the navigation system 200 determines whether there has beena change to the zoom level. If there has been a change to the zoomlevel, then the method 800 returns to step 814, where a different zoomlevel is received. If there has not been a change to the zoom level,then the method 800 proceeds to step 870. At step 870, the navigationsystem 200 determines whether there has been a change to the location ofthe user. If there has been a change to the location of the user, thenthe method 800 returns to step 812, where the location of the user isdetermined. If there has not been a change to the location of the user,then the method 800 proceeds to step 880. At step 880, the navigationsystem 200 determines whether there has been a change to the location ofinterest 110. If there has been a change to the location of interest110, then the method 800 returns to step 810, where a different locationof interest 110 is received. If there has not been a change to thelocation of interest, then the method 800 ends.

FIG. 9 is a block diagram illustrating a computing device 900 configuredto implement one or more aspects of the present invention. As shown,computing device 900 includes a processing unit 902, input/output (I/O)devices 904, and a memory unit 910. Memory unit 910 includes anapplication 912 configured to interact with a database 914.

Processing unit 902 may include a central processing unit (CPU), digitalsignal processing unit (DSP), and so forth. I/O devices 904 may includeinput devices, output devices, and devices capable of both receivinginput and providing output. Memory unit 910 may include a memory moduleor a collection of memory modules. Software application 912 withinmemory unit 910 may be executed by processing unit 902 to implement theoverall functionality of computing device 900, and, thus, to coordinatethe operation of the navigation system 200 as a whole. The database 914may store locations of interest 110, images 112, images associated withreal-time views 116, GPS coordinates, object locations, objectdistances, lookup tables, and other data for generating augmented viewsof locations of interest 110 and acquiring information associated withthe locations of interest 110.

Computing device 900 may be coupled to the camera 210 and the trackingdevice 220. The camera 210 may be configured to capture images of areal-time view 116 of a user. The tracking device 220 may include one ormore sensors, such as one or more cameras and/or depth sensors. Thetracking device 220 may be configured to measure various properties ofthe environment within which user resides, as well as various propertiesassociated with user (e.g., location, position, orientation, and eyegaze direction 223). The tracking device 220 may include any number ofcameras, depth sensors, light sensors, or any other type of sensor.Generally, the tracking device 220 captures sensory data associated withthe environment and sensory data associated with user, and provides thatdata to computing device 900.

Computing device 900 as a whole may be a microprocessor, anapplication-specific integrated circuit (ASIC), a system-on-a-chip(SoC), a mobile computing device such as a tablet computer or cellphone, a media player, and so forth. Generally, computing device 900 isconfigured to coordinate the overall operation of the navigation system200. Any technically feasible system configured to implement thefunctionality of the navigation system 200 falls within the scope of thepresent invention.

In sum, a navigation system determines an image capture location and animage capture direction based on a location of interest and a locationof a user. The navigation system then receives an image of the locationof interest from a database and/or from a camera that is proximate tothe location of interest. The image is based on the image capturelocation and/or the image capture direction. Next, the navigation systemmay process the image by applying a transform based on the location ofthe user. The processed image is then displayed to the user.

One advantage of the techniques described herein is that a user is ableto see through obstructions in the surrounding environment to view alocation of interest. Additionally, the disclosed techniques enable theuser to continue to pay attention to his or her real-time surroundingswhile viewing the location of interest. Accordingly, by overlaying animage of the location of interest with the user's real-time field ofview, navigation efficacy and user safety are increased.

One embodiment of the invention may be implemented as a program productfor use with a computing device. The program(s) of the program productdefine functions of the embodiments (including the methods describedherein) and can be contained on a variety of computer-readable storagemedia. Illustrative computer-readable storage media include, but are notlimited to: (i) non-writable storage media (e.g., read-only memorydevices within a computer such as compact disc read only memory (CD-ROM)disks readable by a CD-ROM drive, flash memory, read only memory (ROM)chips or any type of solid-state non-volatile semiconductor memory) onwhich information is permanently stored; and (ii) writable storage media(e.g., floppy disks within a diskette drive or hard-disk drive or anytype of solid-state random-access semiconductor memory) on whichalterable information is stored.

The invention has been described above with reference to specificembodiments. Persons of ordinary skill in the art, however, willunderstand that various modifications and changes may be made theretowithout departing from the broader spirit and scope of the invention asset forth in the appended claims. For example, although many of thedescriptions herein refer to the user as a driver or passenger of avehicle, persons skilled in the art will appreciate that the systems andtechniques described herein are applicable to other situations (e.g.,non-vehicular) in which generating augmented views of a location ofinterest may enhance user experience and/or user safety. The foregoingdescription and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

Therefore, the scope of embodiments of the present invention is setforth in the claims that follow.

What is claimed is:
 1. A method implemented by a vehicle navigationdevice for generating an augmented view of a location of interest for auser, the method comprising: detecting an eye gaze direction of theuser; determining the location of interest that is directly behind anobstruction at which the user is looking based on the eye gazedirection; selecting an image capture location based on the location ofinterest; selecting an image capture direction based on the imagecapture location; receiving an image associated with the image capturelocation and the image capture direction; processing the image based ona line of sight associated with a location of the user to generate aprocessed image; and displaying the processed image to the user.
 2. Themethod of claim 1, wherein processing the image comprises projecting theline of sight through a surface of a display to determine a transform,and applying the transform to the image.
 3. The method of claim 2,wherein an automobile window comprises the surface of the display. 4.The method of claim 1, further comprising receiving an image of areal-time view associated with the line of sight, and wherein processingthe image comprises comparing the image to the image of the real-timeview, and aligning at least one portion of the image with at least oneportion of the image of the real-time view.
 5. The method of claim 4,wherein the image of the real-time view comprises an obstructed view ofthe location of interest, and the image comprises a substantiallyunobstructed view of the location of interest.
 6. The method of claim 4,wherein processing the image further comprises compositing the imagewith the image of the real-time view.
 7. The method of claim 1, furthercomprising receiving a zoom value from the user, and wherein determiningthe image capture location is further based on the zoom value.
 8. Themethod of claim 1, wherein the location of the user is derived from oneor more geolocation signals.
 9. The method of claim 1, furthercomprising determining an eye gaze direction of the user, and selectingthe location of interest based on the location of the user and the eyegaze direction.
 10. A vehicle navigation system for generating anaugmented view of a location of interest for a user, comprising: aprocessor configured to: receive an eye gaze direction of the user;determine the location of interest that is directly behind anobstruction at which the user is looking based on the eye gazedirection; select a first image capture location based on the locationof interest; select an image capture direction based on the imagecapture location; receive a first image acquired at the first imagecapture location based on the image capture direction; and process thefirst image based on a line of sight associated with a location of theuser to generate a processed image; and a display configured to displaythe processed image to the user.
 11. The system of claim 10, wherein theprocessor is configured to process the first image by projecting theline of sight through a surface of the display to determine a transformand applying the transform to the first image.
 12. The system of claim11, wherein the display is disposed in an automobile window.
 13. Thesystem of claim 10, wherein the processor is further configured toreceive an image of a real-time view associated with the line of sight,and the processor is configured to process the first image by comparingthe first image to the image of the real-time view to determine atransform, and applying the transform to the first image to align atleast one portion of the first image with at least one portion of theimage of the real-time view.
 14. The system of claim 13, wherein theimage of the real-time view comprises an obstructed view of the locationof interest, and the first image comprises a substantially unobstructedview of the location of interest.
 15. The system of claim 10, whereinthe processor is further configured to receive an image of a real-timeview associated with the line of sight, and the processor is configuredto process the first image by aligning at least one portion of the firstimage with at least one portion of the image of the real-time view andcompositing the first image with the image of the real-time view. 16.The system of claim 10, wherein the processor is further configured todetermine a second image capture location based on the location of theuser and the location of interest, and receive a second image acquiredat the second image capture location, and wherein the processor isconfigured to process the first image by combining the first image withthe second image.
 17. The system of claim 16, wherein the first image isassociated with a first perspective of the location of interest, and thesecond image is associated with a second perspective of the location ofinterest.
 18. The system of claim 10, further comprising a trackingdevice configured to determine the eye gaze direction of the user. 19.The system of claim 10, wherein the obstruction comprises a vehicle or abuilding corner.
 20. The system of claim 10, wherein the first imagecomprises a real-time, unobstructed view of the location of interestacquired via a camera mounted on a vehicle.
 21. A non-transitorycomputer-readable storage medium including instructions that, whenexecuted by a processor, cause the processor to generate an augmentedview of a location of interest for a user, by performing the steps of:detecting an eye gaze direction of the user; determining the location ofinterest that is directly behind an obstruction at which the user islooking based on the eye gaze direction; selecting an image capturelocation based on the location of interest; selecting an image capturedirection based on the image capture location; receiving an imageassociated with the image capture location and the image capturedirection; processing the image based on a line of sight associated witha location of the user to generate a processed image; and displaying theprocessed image to the user.